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Abstract 

We study a general class of PageRank optimization problems which 
r"| consist in finding an optimal outlink strategy for a web site subject to 

design constraints. Wc consider both a continuous problem, in which 
one can choose the intensity of a link, and a discrete one, in which in 
, ^ , each page, there are obligatory links, facultative links and forbidden 

links. Wc show that the continuous problem, as well as its discrete vari- 
CN ant when there are no constraints coupling different pages, can both 

be modeled by constrained Markov decision processes with ergodic re- 
ward, in which the webmaster determines the transition probabilities 
of websurfers. Although the number of actions turns out to be ex- 
ponential, we show that an associated polytope of transition measures 
' has a concise representation, from which we deduce that the continuous 

problem is solvable in polynomial time, and that the same is true for 
the discrete problem when there are no coupling constraints. We also 
provide efficient algorithms, adapted to very large networks. Then, wc 
investigate the qualitative features of optimal outlink strategies, and 
: '~j identify in particular assumptions under which there exists a "master" 

rS page to which all controlled pages should point. We report numerical 

^ results on fragments of the real web graph. 
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1 Introduction 



The PageRank introduced by Brin and Page is defined as the invariant 
measure of a walk made by a random surfer on the web graph. When 
reading a given page, the surfer either selects a link from the current page 
(with a uniform probability), and moves to the page pointed by that link, 
or interrupts his current search, and then moves to an arbitrary page, which 
is selected according to given "zapping" probabilities. The rank of a page 
is defined as its frequency of visit by the random surfer. 

The interest of the PageRank algorithm is to give each page of the web 
a measure of its popularity. It is a link-based measure, meaning that it only 
takes into account the hyperlinks between web pages, and not their content. 
It is combined in practice with content-dependent measures, taking into 
account the relevance of the text of the page to the query of the user, in 
order to determine the order in which the answer pages will be shown by the 
search engine. This leads to a family of search methods the details of which 
may vary (and are often not publicly known). However, a general feature 
of these methods is that among the pages with a comparable relevance to 
a query, the ones with the highest PageRank will appear first. 

The importance of optimizing the PageRank, specially for e-business 
purposes, has led to the development of a number of companies offering 
Search Engine Optimization services. We refer in particular the reader to [2j 
for a discussion of the PageRank optimization methods which are used in 
practice. Understanding PageRank optimization is also useful to fight mali- 
cious behaviors like link spamming, which intend to increase artificially the 
PageRank of a web page [3] , [1] . 

The PageRank has motivated a number of works, dealing in particular 
with computational issues. Classically, the PageRank vector is computed 
by the power algorithm [1]. There has been a considerable work on design- 
ing new, more efficient approaches for its computation [5l|6]: Gauss-Seidel 
method [7], aggregation/disaggregation [6] or distributed randomized al- 
gorithms [HI [9]. Other active fields are the development of new ranking 
algorithms [10] or the study of the web graph [11] . 

The optimization of PageRank has been studied by several authors. 
Avrachenkov and Litvak analyzed in [12] the case of a single controlled 
page and determined an optimal strategy. In [13j, Mathieu and Viennot es- 
tablished several bounds indicating to what extent the rank of the pages of 
a (multi-page) website can be changed, and derived an optimal referencing 
strategy in a special unconstrained case: if the webmaster can fix arbitrarily 
the hyperlinks in a web site, then, it is optimal to delete every link pointing 
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outside the web site. To avoid such degenerate strategies, De Kerchove, Ni- 
nove and van Dooren [T3] studied the problem of maximizing the sum of the 
PageRank coordinates in a web site, provided that from each page, there is 
at least one path consisting of hyperlinks and leading to an external page. 
They gave a necessary structural condition satisfied by an optimal outlink 
strategy. In [15], Ninove developed a heuristic based on these theoretical 
results, which was experimentally shown to be efficient. In [16], Ishii and 
Tempo investigated the sensitivity of the PageRank to fragile (i.e. erroneous 
or imperfectly known) web data, including fragile links (servers not respond- 
ing, links to deleted pages, etc.). They gave bounds on the possible variation 
of PageRank and introduced an approximate PageRank optimization prob- 
lem, which they showed to be equivalent to a linear program. In [17 1, (see 
also [TH] for more details), Csaji, Jungers and Blondel thought of fragile 
links as controlled links and gave an algorithm to optimize in polynomial 
time the PageRank of a single page. 

In the present paper, we study a more general PageRank optimization 
problem, in which a webmaster, controlling a set of pages (her web site), 
wishes to maximize a utility function depending on the PageRank or, more 
generally, on the associated occupation measure (frequencies of visit of every 
link, the latter are more informative). For instance, the webmaster might 
wish to maximize the number of clicks per time unit of a certain hyperlink 
bringing an income, or the rank of the most visible page of her site, or the 
sum of the ranks of the pages of this site, etc. We consider specifically two 
versions of the PageRank optimization problem. 

We first study a continuous version of the problem in which the set of 
actions of the webmaster is the set of admissible transition probabilities of 
websurfers. This means that the webmaster, by choosing the importance of 
the hyperlinks of the pages she controls (size of font, color, position of the 
link within the page), determines a continuum of possible transition prob- 
abilities. Although this model has been already proposed by Nemirovsky 
and Avrachenkov [T9], its optimization does not seem to have considered 
previously. This continuous version includes rather realistic constraints: for 
instance, the webmaster may start from a "template" or "skeleton" (given by 
designers), and be allowed to modify this skeleton only to a limited extent. 
Moreover, we shall allow coupling constraints between different pages (for 
instance, the rank of one page may be required to be greater than the rank 
of another page, constraints involving the sum of the pageranks of a subset 
of pages are also allowed, etc.). 

Following [16l [T7j, we also study a discrete version of the problem, in 
which in each page, there are obligatory links, facultative links and forbidden 
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links. Then, the decision consists in selecting the subset of facultative links 
which are actually included in the page. 

We show that when there are no coupling constraints between different 
pages and when the utility function is linear, the continuous and discrete 
problems both can be solved in polynomial time by reduction to a linear 
program (our first main result, Theorem |4]). When specialized to the dis- 
crete problem, this extends Theorem 1 of pTT], which only applies to the case 
in which the utility function represents the PageRank of a single page. The 
proof of Theorem |4] relies on the observation that the polytope generated 
by the transition probability measures that are uniform on some subsets 
of pages has a concise representation with a polynomial number of facets 
(Theorem [T]). This leads us to prove a general result of independent in- 
terest concerning Markov decision processes with implicitly defined action 
sets. We introduce the notion of well-described Markov decision processes, 
in which, although there may be an exponential number of actions, there is a 
polynomial time strong separation oracle for the actions polytope (whereas 
the classical complexity results assume that the actions are explicitly enu- 
merated |20j ) . We prove in Theorem [sj as an application of the theory of 
Khachiyan's ellipsoid method (see [21])) that the ergodic control problem for 
well-described Markov decision process is polynomial time solvable (even in 
the multi-chain framework). Then, Theorem |4] follows as a direct corol- 
lary. We note that maximization or separation oracles have been previously 
considered in dynamic programming for different purposes (dealing with 
unnkown parameters [221 123j , or approximating large scale problems [24J ) . 

Proposition [7] yields a fixed point scheme with a contraction rate inde- 
pendent of the number of pages. Indeed, the contraction rate depends only 
on the "damping factor" (probability that the user interrupts his current 
search). Therefore, this problem can be solved efficiently for very large in- 
stances by Markov decision techniques. Our results show that optimizing 
the PageRank is not much more difficult than computing it, provided there 
are no coupling constraints: indeed. Proposition [9] shows that by compari- 
son, the execution time is only increased by a logn factor, where n is the 
number of pages. Note that the Markov decision process which we construct 
here is quite different from the one of [17J, the latter is a stochastic shortest 
path problem, whose construction is based on a graph rewriting technique, 
in which intermediate (dummy) nodes are added to the graph. Such nodes 
are not subject to damping and therefore, the power iteration looses its uni- 
form contraction. In our approach, we use a more general ergodic control 
model, which allows us to consider a general linear utility function, and 
avoids adding such extra nodes. Experiments also show that the present 
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approach leads to a faster algorithm (Section |7.2[ ). 

We also study the continuous problem with general (linear) coupling con- 
straints, and show that the latter can also be solved in polynomial time by 
reduction to a constrained ergodic control problem. Proposition [13] yields 
an algorithm to solve the PageRank optimization problem with coupling 
constraints, which scales well if the number of coupling constraints remains 
small. The resolution uses Lagrangian relaxation and convex programming 
techniques like the bundle method. There is little hope to solve efficiently, in 
general, the discrete problem with general coupling constraints since Csaji, 
Jungers and Blondel have proved in [T7] that the discrete PageRank opti- 
mization problem with mutual exclusion constraints is NP-complete. Nev- 
ertheless, we develop a heuristic for the discrete PageRank optimization 
problem with linear coupling constraints, based on the optimal solution of 
a relaxed continuous problem (Section 7.3). On test instances, approximate 
optimality certificates show that the solution found by the heuristic is at 
most at 1.7% of the optimum. 

Using the concept of mean reward before teleportation, we identify in 
Theorem [5] (our second main result) assumptions under which there exists 
a "master" page to which all controlled pages should point. The theorem 
gives an ordering of the pages such that in loose terms, the optimal strategy 
is at each page to point to the allowed pages with highest order. The struc- 
ture of the obtained optimal website is somehow reminiscent of Theorem 12 
in |14j . but in |14] . there is only one constraint: the result is thus differ- 
ent. When the problem has coupling constraints, the mean reward before 
teleportation still gives information on optimal strategies (Theorem [g]). 

We report numerical results on the web site of one of the authors (in- 
cluding an aggregation of surrounding pages) as well as on a fragment of the 
web (4.10^ pages from the universities of New Zealand). 

We finally note that an early Markov Decision Model for PageRank op- 
timization was introduced by Bouhtou and Gaubert in 2007, in the course 
of the supervision of the student project of Vlasceanu and Winkler |25j . 

The paper is organized as follows. In Section [2| we introduce the general 
PageRank optimization problem. In Section [3j we give a concise description 
of the polytope of uniform transition probabilities. In Section [4j we show 
that every Markov decision process which admits such a concise description 
is polynomial time solvable (Theorem [s]) , and we deduce as a corollary our 
first main result. Theorem [4| Section 4.3 describes an efficient fixed point 
scheme for the resolution of the PageRank optimization problem with local 
constraints. In Section [5| we give the "master page" Theorem (Theorem [s]). 
We deal with coupling constraints in Section [6} We give experimental results 



5 



on real data in Section [7l 



2 PageRank optimization problems 
2.1 Google's PageRank 

We first recall the basic elements of the Google PageRank computation, 
see [1] and [6j for more information. We call web graph the directed graph 
with a node per web page and an arc from page i to page j if page i contains 
a hyperlink to page j. We identify the set of pages to [n] := {1, . . . , n}. 

Let Ni denote the number of hyperlinks contained in page i. Assume 
first that Ni > 1 for all i G [n], meaning that every page has at least one 
outlink. Then, we construct the n x n stochastic matrix S, which is such 
that 



This is the transition matrix of a Markov chain modeling the behavior of a 
surfer choosing a link at random, uniformly among the ones included in the 
current page and moving to the page pointed by this link. The matrix S 
only depends of the web graph. 

We also fix a row vector z £ M", the zapping or teleportation vector, 
which must be stochastic (so, Ylje[n] ~ 1)' together with a damping factor 
a G [0, 1] and define the new stochastic matrix 



where e is the (column) vector in M"' with all entries equal to 1. 

Consider now a Markov chain {Xt)t>o with transition matrix P, so that 
for all i,j G [n], F{Xt-\-i = j\Xt = i) = Pij. Then, Xt represents the position 
of a websurfer at time t: when at page i, the websurfer continues his current 
exploration of the web with probability a and moves to the next page by 
following the links included in page i, as above, or with probability 1 — a, 
stops his current exploration and then teleports to page j with probability Zj . 

When some page i has no outlink, Ni = 0, and so the entries of the 
ith. row of the matrix S cannot be defined according to ([T]). Then, we set 
Sij := Zj. In other words, when visiting a page without any outlink, the 
websurfer interrupts its current exploration and teleports to page j again 
with probability Zj . It is also possible to define another probability vector Z 
(different from z) for the teleportation from these "dangling nodes" . 
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if page j is pointed to from page i 
otherwise 



(1) 



P = aS + (1 — a)ez 
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The PageRank vr is defined as the invariant measure of the Markov chain 
{Xt)t>o representing the behavior of the websurfer. This invariant measure 
is unique if a < 1, or if P is irreducible. 

Typically, one takes a = 0.85, meaning that at each step, a websurfer 
interrupts his current search with probability 0.15 ~ 1/7. The advantages 
of the introduction of the damping factor and of the teleportation vector 
are well known. First, it guarantees that the power algorithm converges to 
the PageRank with a geometric rate a independent of the size (and other 
characteristics) of the web graph. In addition, the teleportation vector may 
be used to "tune" the PageRank if necessary. By default, z = e^/n is the 
uniform stochastic vector. We will assume in the sequel that a < 1 and 
Zj > for all j £ [n], so that P is irreducible. 

The graph on Figure [T] represents a fragment of the web graph. We ob- 
tained the graph by performing a crawl of our laboratory with 1500 pages. 
We set the teleportation vector in such a way that the 5 surrounding insti- 
tutional pages are dominant. The teleportation probabilities to these pages 
were taken to be proportional to the PageRank (we used the Google Tool- 
bar, which gives a rough indication of the PageRank, on a logarithmic scale). 
After running the PageRank algorithm on this graph, we found that within 
the controlled site, the main page of this author has the biggest PageRank 
(consistently with the results provided by Google search). 

2.2 Optimization of PageRank 

The problem we are interested in is the optimization of PageRank. We study 
two versions of this problem. In the continuous PageRank Optimization 
problem, the webmaster can choose the importance of the hyperlinks of the 
pages she controls and thus she has a continuum of admissible transition 
probabilities (determined for instance by selecting the color of a hyperlink, 
the size of a font, or the position of a hyperlink in a page). This continuous 
model is specially useful in e-business applications, in which the income 
depends on the effective frequency of visit of pages by the users, rather 
than on its approximation provided by Google's pagerank. The Continuous 
PageRank Optimization Problem is given by: 

max{C/(7r,P) ; vr = ttP, vr E P G V} (2) 

7r,P 

Here, E„ := {x G M" | Xj > 0,Vi G [n]; X]ie[n] ^« ~ ^} simplex 
of dimension n, C/ is a utility function and is a set representing the set 
of all admissible transition probability matrices. We denote by Pj_. the 
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Figure 1: The web site of one of the authors (colored) and the surrounding 
sites (white). This 1500-page fragment of the web is aggregated for presen- 
tation, using the technique described in [6]. The sizes of the circles follow 
the log of their PageRank. 
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ith row of a matrix P. We shall distinguish local constraints, which can 
be expressed as Pi,. G Vi, where Vi C Sn is a given subset, and global 
constraints, which couple several vectors Pi^.. Thus, local constraints only 
involve the outlinks from a single page, whereas global constraints involve 
the outlinks from different pages. We shall consider the situation in which 
each Vi is a polytope (or more generally an effective convex set). 

If we restrict our attention to Google's PageRank (with uniform tran- 
sition probabilities), we arrive at the following combinatorial optimization 
problem. For each page i, as in [16] and [IT], we partition the set of po- 
tential links into three subsets, consisting respectively of obligatory 
links Oi, prohibited links Xi and the set of facultative links Ti- Then, for 
each page i, we must select the subset Jj of the set of facultative links 
Ti which are effectively included in this page. Once this choice is made 
for every page, we get a new webgraph, and define the transition matrix 
S = S{Ji, . . . , Jn) as in ([T]). The matrix after teleportation is also defined 
as above by P{Ji, ■ ■ ■ , Jn) ■= aS{Ji, . . . , Jn) + (1 — a)ez. Then, the Discrete 
PageRank Optimization Problem is given by: 



max{[/(7r, P) ; vr = vrP, vr G S„, P = P{Ji, ...,Jn), Ji ^ J^i, i ^ [n]} 



Remark 1. Problem ([3]) is a combinatorial optimization problem: if there 
are pi facultative links in page i, the decision variable, (Ji, . . . , Jn), takes 2^ 
values, where p = pi + ■ ■ ■ + Pn- 

We shall be specially interested in the modeling of an income propor- 
tional to the frequency of clicks on some hyperlinks. Let rjj be a reward 
per click for each hyperlink {i,j). The latter utility can be represented by 
the following linear utility function, which gives the total income: 



Unless stated otherwise, we will consider the total income linear utility in 
the sequel. 

Remark 2. The problem of maximizing the total PageRank of a web site 
(sum of the PageRanks of its pages) is obtained as a special case of Q. 
Indeed, if this web site consists of the subset of pages / C [n], one can set 
= X/(0'^^'i S ['^l' where xi is the characteristic function of / (with 
value 1 if i G / and otherwise). Then U{'k,P) = Yli'^iYlj — 



7T,P 



(3) 




(4) 
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Remark 3. Note that the general form of the utility function assumes that 
we receive the same instantaneous reward rij when the surfer follows the 
hyperlink and when the surfer stops the current exploration at page 
i to teleport to page j. There is no loss of generality in assuming that 
it is so: assume that the surfer produces a reward of r'^ j when he follows 
the hyperlink and when he teleports to page j. Using the fact that 

EisH r'ijZj = J2je[n] E«e[n] r'i^iZiPij and P = aS + {l-a)ez, we show that 
a EijeN ''iJ^'i^iJ = Ei,j&[n]Kj - (1 - a) Eie[n] r'i,iZi)T^iPi,r We then only 
need to set r^j = r'^ - - {I - a) Y.i&[n] 

We shall restrict our attention to situations in which tt is uniquely defined 
for each admissible transition matrix P £ V (recall that this is the case in 
particular when a < 1). Then the utility [/ is a function of P only. 

Alternatively, it will be convenient to think of the utility as a function 
of the occupation measure p = {Pi,j)i,j£[n]- The latter is the stationary 
distribution of the Markov chain {xt-i,xt). Thus, pij gives the frequency of 
the move from page i to page j. The occupation measure p is a probability 
measure and it satisfies the flow relation, so that 

Pi,j > 0, Vi,j G [n] , ^ pij = 1 , ^ pk^i = ^ pij, Vi € [n] . 

i,j&[n] fce[n] je[n] 

(5) 

The occupation measure may also be thought of as a matrix. Hence, we 
shall say that p is irreducible when the corresponding matrix is irreducible. 

The occupation measure p can be obtained from the invariant measure tt 
and the stochastic matrix P by pij = TTiPij,yi,j £ [n] and, conversely, the 
invariant measure n can be recovered from p by tTj = J2je[n] PiJ^"^^ ^ iM- 

The map / which determines the stochastic matrix P from the occupa- 
tion measure is given by: 

P = /(p), P,,. = ^!^, Vi,je[n]. (6) 

Proposition 1. The function f defined by ^ sets up a birational transfor- 
mation between the set of irreducible occupation measures (irreducible ma- 
trices satisfying ^ ) and the set of irreducible stochastic matrices. In par- 
ticular, the Jacobian of f is invertible at any point of the set of irreducible 
occupation measures. 

Proof. As TT is uniquely defined, its entries are a rational function of the 
entries of P (for instance, when P is irreducible, an explicit rational expres- 
sion is given by Tutte's Matrix Tree Theorem [26]). The invertibility of the 
Jacobian follows from the birational character of /. □ 
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This bijective correspondence will allow us to consider the occupation 
measure, rather than the stochastic matrix P, as the decision variable. Note 
that the utility function can be written as a linear function in terms of the 
occupation measure: U{tt,P) = X^jjeH Pi,Fi,j- 

2.3 Design constraints of the webmaster 

We now model the possible modifications made by the webmaster, who 
may be subject to constraints imposed by the designer of the web site (the 
optimization of the PageRank should respect the primary goal of the web 
site, which is in general to offer some content). We thus describe the set V 
of admissible transition probabilities of ([2]). 

Proposition 2. Assume that V = Hieln]'^*' ^^'^^ /^'^ '^^^ ^ ^ N; '^i 
a closed convex and that every matrix P & V is irreducible. Then, the set 
TZ of occupation measures arising from the elements of V is also a closed 
convex set. Moreover, if every Vi is a polytope, then so is TZ. 

Proof. For all i £ [n], Vi is a closed convex set and so it is the intersection 
of a possibly infinite family of hyperplanes {Hf'^)i^L. Every element P of 
riieH ™ust satisfy the following inequalities, one for each Hf^: 

^agp,,,-<6f\ ViG[n],V/GL (7) 

Formulating these equalities in terms of the occupation measure p thanks to 

^ij ~ Y"^''^ — ^iid Proposition ll and rewriting Inequalities (iTj) in the form 
' l^ji Pi,j' |_| ■— ' 

a^]p^,j < hf Yl P^^k, Vi €[n],yiGL (8) 

je[n] ke[n] 

we see that p satisfies a family of constraints of the form ([8|, together with 
the inequalities ([s]). Thus, IZ is defined as the intersection of half-spaces and 
so, it is closed and convex. 

The same argument shows that if for all i £ [n], Vi is a polytope, so is 
TZ. □ 

We next list some concrete examples of such inequalities. 
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Skeleton constraints Imagine that a designer gave a skeleton or template 
for page i. The latter may include a collection of mandatory sites to be 
pointed by page i. We shall abstract the skeleton by representing it by 
a fixed probability vector q £ giving the transition probabilities if no 
further hyperlinks are added. Assume now that the webmaster is allowed 
to modify the page for optimization purposes, as long as the hyperlinks 
she adds do not overtake the initial content of the web site. This can be 
modeled by requiring that no hyperlink included in the skeleton looses a 
proportion of its weight greater than fi. Such constraints can be written as 
Pi,j > a{l - fj.)qj + (1 - a)zj, Mj £ [n]. 

Linear coupling constraints Constraints like the presence of specific 
outlinks somewhere on the pages of the website are non-local. Such con- 
straints cannot be written simply in terms of the stochastic matrix P (be- 
cause adding conditional probabilities relative to different pages makes little 
sense) but they can be written linearly in terms of the occupation measure 
Si je[n] ^hjPi,j — ^' where the coefficients Uij and b are given. 
These constraints include for instance coupling conditional probability 
constraints, which can be written as: Yli^i ji^j Pi,j — ^Sie/ fce[n] /^*>fc- This 
means that the probability for the random surfer to move to set J, given 
that he is now in set /, should not be smaller than 6. 



Combinatorial constraints In the discrete problem, one may wish to 
set combinatorial constraints like demanding the existence of a path be- 
tween two pages or sets of pages [l3| , setting mutual exclusion between two 
hyperlinks [17] or limiting the number of hyperlinks [17] . Such constraints 
may lead to harder combinatorial problems, the solution of which is how- 
ever made easier by the polynomial-time solvability of a relaxed continuous 
problem (Section 7.3). 



3 The polytope of uniform transition measures 

In this section, we show that the polytope of uniform transition measures 
admits a concise representation (Theorem [T]) . The vertices of this polytope 
represent the action space of the Discrete PageRank Optimization prob- 
lem ([3]). Theorem [1] is a key ingredient of the proof of the polynomial time 
character of this problem which will be given in the next section. 

We consider a given page i and we study the set of admissible transition 
probabilities from page i. With uniform transitions, this is a discrete set 
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that we denote Pj. For clarity of the explanation, we will write xj instead 
of Si J and write the proofs in the case a = 1. To get back to a < 1, we 
use the relation Pjj = aSij + (1 — a)zj (see Remark [s] at the end of this 
section). 

We partition the set of links from page i as the set of obligatory links 
Oi, the set of prohibited links li and the set of facultative links J-i. Then, 
depending on the presence of obligatory links. 

Pi = {g G S" I C supp(g) QdU J^i, 

q uniform probability measure on its support} (9) 

or if Oj = 0, it is possible to have no link at all and then to teleport with 
probability vector Z: 

Pi = {g G I supp(g) C Ti, 

q uniform probability measure on its support} U {Z} . 

We study the polytope co (Pi), the convex hull of the discrete set Pj. 
Although it is defined as the convex hull of an exponential number of points, 
we show that it has a concise representation. 

Theorem 1. // page i has at least one obligatory link, then the convex 
hull of the admissible discrete transition probabilities from page i, co(Pi), is 
the projective transformation of a hypercube of dimension and, for any 
choice of jo G Oi, it coincides with the polytope defined by the following set 
of inequalities: 

\/j £ li , = ^ -^i 1 ^ ^jo (10a) 

Vj G Oi \ {jo} , Xj = Xj, yj G , X,- > (10b) 

Y.x, = l (10c) 

ie[n] 



Proof. Let Si be the polytope defined by Inequalities (10). 

(Pi C 5i): Let q a probability vector in Pj: q \s & uniform probability 
measure on its support and Oi C supp(g) Oi U Ti. As for all j in J^j, 
1j - |supp(g)| = ^io' Q verifies the equalities. 

(extr(5i) C Pj): Let us consider an extreme point x of Si. Inequal- 
ities (10b) and ( 10a[ ) cannot be saturated together at a given coordinate 
j G J-i because, if it were the case, then we would have Xj, = and thus 
X = 0, which contradicts J2je[n] — ^■ 
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We have 1 + + lOjl — 1 independent equalities so the polytope is of 
dimension \ J-i\. To be an extreme point, x must thus saturate \ J-i\ inequal- 
ities. At every j in Ti, Inequalities (10b) and (10a) cannot be saturated 



simultaneously (see the previous paragraph), so the only way to saturate 
\J-i\ inequalities is to saturate one of (10b) or (10a) at every j in J^j. Fi- 
nally, X can only take two distinct values, which are and Xj^^ = |supp(^)| ■ 
it is a uniform probability on it support. 

We then show that Si is the projective transformation ([27], Section 2.6 
for more background) of the hypercube H defined by the following set of 
inequalities: 

{Vj G Xi,Xj = ; Vj G Oi,Xj = 1 ; Vj G 7-^,0 < Xj < 1} . 

As Oj 7^ 0, is embedded in the affine hyperplane {-^ G M^lXj^, = 1}. 
We can then construct the homogenization of H, homog(-ff), which is the 
pointed cone with base H (see [27] for more details). Finally Si is the cross- 

□ 



section of homog(i?) with the hyperplane {x G M"| ^ 



je[n\ 



!}■ 



The result of the theorem implies in particular that co(2?j) is combina- 
torially equivalent to a hypercube, ie. that their face lattices are isomor- 
phic 



Figure 2: Projection of the polytope of uniform transition measures with 
one obligatory link {\Oi\ = 1) and three facultative links {\Ti\ = 3). 

The next result concerns the case in which a page may have no outlink: it 
is necessary to consider this special case because then the websurfer teleports 
with probability Zi to page i. 

Proposition 3. If page i has no obligatory link and if there exists k G li 
such that Zk > 0, then co(Pj) is a simplex of dimension \ J-i\ defined by the 
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following set of inequalities: 



i6[n] 

z z 

Vj {k} , Xj = Xfc , Vj G J"i , > ^Xk (lib) 

Proof. The proof follows the same sequence of arguments as the proof of 
Theorem [T] We just need to adapt it to Inequalities (11). □ 

Proposition 4. If page i has no obligatory link and if for all A; € Xj, Zj. = 0, 
then co(2?j) is the usual simplex of dimension \ J-i\ — 1 with Xj = 0, \/j G Xj. 

Proof. The extreme points of this simplex are clearly admissible discrete 
transition probabilities and the polytope contains every admissible discrete 
transition probabilities. □ 

Remark 4. When there is no obligatory link, most of the admissible discrete 
transition probabilities are not extreme points of the polytope. 

Remark 5. If we want to work with Vi, the polytope of transition proba- 
bilities with damping factor a, we only need the relation Vi = aSi + {l — a)z 
to get the actual inequalities. For instance, xj = xj^ remains but Xj > 
becomes xj > (1 — a)zj. 



4 Solving the PageRank Optimization Problem with 
local constraints 

4.1 Reduction of the PageRank Optimization Problem with 
local constraints to Ergodic Control 

We next show that the continuous and discrete versions of the PageRank 
optimization reduce to ergodic control problems in which the action sets are 
defined as extreme points of concisely described polyhedra. 

A finite Markov decision process is a 4-uple (I, r) where / is a 

finite set called the state space; for all i £ I, Ai is the finite set of admissible 
actions in state i; p : I x Uig/({i} x Ai) — )• ]R_(_ is the transition law, so that 
p{j\i,a) is the probability to go to state j form state i when action a £ Ai 
is selected; and r : Uig/({i} x Ai) — )■ M is the reward function, so that r(i, a) 
is the instantaneous reward when action a is selected in state i. 

Let Xf £ I denote the state of the system at the discrete time t > 0. A 
deterministic control strategy is a sequence of actions (ft)t>o such that for 
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all t > 0, is a function of the history hf = {Xq, z^q, • • • , Xt-i, i^t-i, Xt) and 
ut G Ax,. Of course, F{Xt+i = j\Xt, vt) = p{j\Xt, ut),^3 e N, Vt > 0. More 
generally, we may consider randomized strategies u where vt is a probability 
measure on A strategy v is stationary (feedback) if there exists a 

function i> such that for all t > 0, vt{ht) = i^{Xt). 

Given an initial distribution /i representing the law of Xq, the average 
cost infinite horizon Markov decision problem, also called ergodic control 
problem, consists in maximizing 

1 ^"^ 

\hnh£-E{Y,riXt,ut)) (12) 

where the maximum is taken over the set of randomized control strategies i^. 
Indeed, the supremum is the same if it is taken only over the set of random- 
ized (or even deterministic) stationary feedback strategies (Theorem 9.1.8 
in |28| for instance). 

A Markov decision process is unichain if the transition matrix corre- 
sponding to every stationary policy has a single recurrent class. Otherwise 
it is multichain. When the problem is unichain, its value does not depend 
on the initial distribution whereas when it is not, one may consider a vector 



{9i)iei where gi represents the value of the problem (12) when starting from 
state i. 

Proposition 5. If there are only local constraints, ie. V = HieH 
for all i £ [n], Vi is a polytope and if the utility function is an income 
proportional to the frequency of clicks then the continuous PageRank 
Optimization problem <^ is equivalent to the unichain ergodic control prob- 
lem with finite state [n], finite action set extr(Pj) in every state i, transition 
probabilities p{j\i, a) = aj and rewards r{i, a) = J2je[n] '''hj^j- 

Proof. As a < 1, a £ Vi implies Ofc > for all k. Thus the problem defined 
in the proposition is unichain. Randomized stationary strategies are of the 
form ut = i^{Xt) for some function v' sending i € [n] to some element of 
Vi = co(extr(Pj)). To such a strategy is associated a transition matrix P 
of the websurfer, obtained by taking Pi^. = z/(i) and vice versa. Thus, the 
admissible transition matrices of the websurfer are admissible stationary 
feedback strategies. 

Moreover, the ergodic theorem for Markov chains shows that when such 
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a strategy is applied, 



T-1 , T-1 



and so, the objective function of the ergodic control problem is precisely the 
total income. □ 

Proposition 6. The following dynamic programming equation: 

Wi + ip = maxz^(rj^. + w) , Vi G [n] (13) 

has a solution w £ M" and -0 G M. The constant tp is unique and is the value 
of the PageRank Optimization problem p[). An optimal strategy is obtained 



by selecting for each state i a maximizing v £ Vi in (13). The function w 
is often called the bias or the potential. 

Proof. Theorem 8.4.3 in [28j applied to the unichain ergodic control problem 
of Proposition [5] implies the result of the proposition but with Vi replaced 
by extr('Pj). But as the expression which is maximized is affine, using Vi or 
extr('Pj) yields the same solution. □ 

Theorem 2. The discrete PageRank Optimization problem ^ is equivalent 
to a continuous PageRank Optimization problem ^ in which the action set 
Vi is defined by one of the polytopes described in Theorem^ or Proposition^ 
or^ depending on the presence of obligatory links. 

Proof. Arguing as in the proof of Proposition [Sj we get that the discrete 
PageRank Optimization problem ([s]) is equivalent to an ergodic control prob- 
lem with state space [n], in which the action set in state i is the discrete set 
defined in ([9]), and the rewards and transition probabilities are as in this 
proposition. The optimal solutions of the discrete PageRank Optimization 
problem coincide with the optimal stationary deterministic strategies. The 



analog of Equation ( 13 ) is now 



Wi + ijj = max i^(rj_. + w) (14) 

!^Sco(I5i) 

where co(Pj) is the convex hull of the set Vi, i.e the polytope described in 
either Theorem [T] or Proposition |3] or |4j The polytope co(Dj) gives the tran- 
sition laws in state i corresponding to randomized strategies in the former 
problem. Hence, the control problems in which the actions sets are Vi or 
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co(T>i) have the same value. Moreover, an optimal strategy of the problem 



with the latter set of actions can be found by solving ( 14 ) and selecting a 



maximizing action u in ( 14 ). Such an action may always be chosen in the set 
of extreme points of co(Pi) and these extreme points belong to (beware 
however that some points of T>i may be not extreme). □ 

4.2 Polynomial time solvability of well-described Markov de- 
cision problems 

We have reduced the discrete and continuous PageRank Optimization prob- 
lems to ergodic control problems in which the action sets are implicitly de- 
fined as the sets of extreme points of polytopes. Theorem 1 in [20] states 
that the ergodic control problem is solvable in polynomial time. However, in 
this result, the action sets are defined explicitly, whereas polynomial means, 
as usual, polynomial in the input length (number of bits of the input). Since 
the input includes the description of the actions sets, the input length is al- 
ways larger than the sum of the cardinalities of the action sets. Hence, this 
result only leads to an exponential bound in our case (Remark [T]). 

However, we next establish a general result. Theorem [3] below, showing 
that the polynomial time solvability of ergodic control problems subsists 
when the action sets are implicitly defined. This is based on the combi- 
natorial developments of the theory of Khachiyan's ellipsoid method, by 
Groetschel, Lovasz and Schrijver |21j . We refer the reader to the latter 
monograph for more background on the notions of strong separation oracles 
and well described polyhedra. 

Definition 1 (Def. 6.2.2 of [21j). We say that a polyhedron B has facet- 
complexity at most (j) if there exists a system of inequalities with rational 
coefficients that has solution set B and such that the encoding length of 
each inequality of the system (the sum of the number of bits of the rational 
numbers appearing as coefficients in this inequality) is at most (p. 

A well-described polyhedron is a triple {B; n, (j)) where B £ M" is a poly- 
hedron with facet-complexity at most (p. The encoding length of B is by 
definition n + 0. 

Definition 2 (Problem (2.1.4) of [21]). A strong separation oracle for a set 
K is an algorithm that solves the following problem: given a vector y, decide 
whether y G K or not and if not, find a hyperplane that separates y from K; 
i.e., find a vector c such that c^y > max{c"^x,x G K}. 

Inspired by Definition [T| we introduce the following notion. 



18 



Definition 3. A finite Markov decision process {I,{Ai)i^j,p,r) is well- 
described if for every state z G /, we have Ai C M.^' for some Li G N, 
if there exists (/> S N such that the convex huh of every action set Ai 
is a well-described polyhedron [Bi^Li^cj)) with a polynomial time strong 
separation oracle, and if the rewards and transition probabilities satisfy 
'r{ha) = Ylie[L,]^l^i and p{j\i,a) = T.ie[L,]^lQi,j^ ^hJ G /, Va G Ai, 
where R\ and Q\ j are given rational numbers, for and / G [Li]. 

The encoding length of a well-described Markov decision process is by 
definition the sum of the encoding lengths of the rational numbers Q[ j and 
R[ and of the well-described polyhedra Bi . 

The situation in which the action spaces are given as usual in extension 
(by listing the actions) corresponds to the case in which Ai is the set of 
extreme points of a simplex S l. . The interest of Definition[3]is that it applies 
to more general situations in which the actions are not listed, but given 
implicitly by a computer program deciding whether a given element of M^* 
is an admissible action in state i (the separation oracle) . An example of such 
a separation oracle stems from Theorem [T] here, a potential (randomized) 
action is an element of M", and to check whether it is admissible, it suffices 
to check whether one of the inequalities in ( |10[ ) is not satisfied. 

Theorem 3. The average cost infinite horizon problem for a well-described 
(multichain) Markov decision process can be solved in a time polynomial in 
the input length. 

Proof. We shall use the notations of Definition|3j Consider the polyhedron Q 
consisting of the couples of vectors (f , g) G x M.^ satisfying the constraints 

fifj > ^ ^ aiQ[jgj , \/i £ I,ae Ai 

^enelL, ^^^^ 

Vi + gi> aiR\ + (^iQijVj > yi £ I,a £ Ai . 

ie[L,] jai ze[L,] 

Theorem 9.3.8 in [28j implies that the average cost problem reduces to min- 
imizing the linear form {v,g) i— ?■ '^j^j gj over Q. Every optimal solution 
{v,g) of this linear program is such that gj is the optimal mean payment 
per time unit starting from state j. We recover optimal strategies of the 
ergodic problem through dual optimal solution of the linear program. 

By Theorem 6.4.9 in [21], we know that a linear program over a well- 
described polyhedron with a polynomial time strong separation oracle is 
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polynomial time solvable. Moreover, Theorem 6.5.14 in [2T] asserts that we 
can find a dual optimal solution in polynomial time. 

Let us construct such an oracle for Q. Given a point {g,v) S Q*^ x 
Q", compute for all i G /: maXagco(yl,) E/e[L,] "KEje/ ^Ij^i) - and 

Eze[L,] "^K-^i + Eje/ QiJ^j) -Vi- Qi. Those problems are lin- 
ear problems such that, by hypothesis, we have a polynomial time strong 
separation oracle for each of the well-described polyhedral admissible sets 
Bi = co(^j). Thus they are polynomial time solvable. If the 2n linear pro- 
grams return a nonpositive value, then this means that {g, v) is an admissible 



point of ( 15 ). Otherwise, the solution a of any of those linear programs that 
have a negative value yields a strict inequality gi < J2jei'^i&[Li]^iQi j9j 

or Vi+ gi < ^i(z[L.] aiR\ + Yljai Z]/e[Li] ^iQ\,j'"i- both cases, the corre- 
sponding inequality determines a separating hyperplane. 

To conclude the proof, it remains to check that the facet complexity of 
the polyhedron Q is polynomially bounded in the encoding lengths of the 
polyhedra Bi and the rationals R\ and Q\j- Since the a/'s appear linearly in 



the constraints (15), these constraints hold for all a G if and only if they 
hold for all a ^ Bi ox equivalently, for all extreme points of Bi. The result 
follows from Lemma 6.2.4 in [21j, which states that the encoding length of 
any extreme point of a well-described polyhedron is polynomially bounded 
in the encoding of the polyhedron. □ 

Remark 6. This argument also shows that the discounted problem is poly- 
nomial time solvable. 

As a consequence of Theorems [2] and [3j we get 

Theorem 4. If there are only local constraints, if the utility function is a ra- 
tional total income utility Q and if the teleportation vector and damping 
factor are rational, then the discrete problem ^ can be solved in polyno- 
mial time and the continuous problem ^ with well-described action sets 
(Definition^ can also be solved in polynomial time. 

Proof. Thanks to Theorem [2j solving the continuous PageRank Optimiza- 
tion problem also solves the discrete PageRank Optimization problem. In 
addition, the coefficients appearing in the description of the facets of the 
polytopes of uniform transition measures are either 1, Zj or a and there are 
at most two terms by inequality (cf Section |3]). This implies that these poly- 
topes are well-described with an encoding length polynomial in the length 
of the input. Note also that we can find in polynomial time a vertex optimal 
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solution of a linear program as soon as its feasible set is a polytope as it is 
the case here (Lemma 6.5.1 in [21J). 

By Proposition [5j the ergodic control problem associated to a continuous 
PageRank Optimization problem with well-described action sets satisfies the 
conditions of Theorem js] with I = [n], Li = [n], QI j = 6ji and R\ = ri^i for 
i^j G [n],/ G Li. Thus it is polynomial time solvable. □ 

Theorem [3] is mostly of theoretical interest, since its proof is based on 



the ellipsoid algorithm, which is slow. We however give in Section 4.3 a fast 
scalable algorithm for the present problem. 

Example 1. Consider again the graph from Figure [T| and let us optimize 
the sum of the PageRank scores of the pages of the site (colored) . Assume 



that there are only local skeleton constraints (see Section 2.3): each page 
can change up to 20 % of the initial transition probabilities. The result is 
represented in Figure [3} 

Example 2. We now consider a discrete Pagerank optimization problem 
starting from the same graph. We set obligatory links to be the initial links 
and we represent them on the adjacency matrix in Figure |4] by squares. 
Facultative links are all other possible links from controlled pages. 

4.3 Optimizing the PageRank via Value iteration 

The PageRank optimization is likely not to be applied to the world wide 
web, but rather to a fragment of it, consisting of a web site (or of a col- 
lection of web sites of a community) and of related sites (see Remark 14 in 
Section [5]) However, even in such simplified instances, the number of design 
variables may be large, typically between thousands and millions. Hence, 
it is desirable to have scalable algorithms. We next describe two methods, 
showing that the optimization problem is computationally easy when there 
are no coupling constraints: then, optimizing the PageRank is essentially 
not more expensive than computing the PageRank. 

Proposition 7. Let T he the dynamic programming operator M" — )• M" 
defined by 

Ti{w) = max oiv{ri^. + w) + {1 — a)z ■ ri^. , Vz G [n] . 
The map T is a- contracting and its unique fixed point w is such that 



{w, {l—a)zw) is solution of the ergodic dynamic programming equation (13). 
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Figure 3: Web graph of Figure [T] optimized under local skeleton constraints. 
The optimal strategy consists in linking as much as possible to page "c" 
(actually, the page of a lecture), up to saturating the skeleton constraint. 
This page gains then a PageRank comparable to the one of the main page. 
The sum of the PageRank scores has been increased by 22.6%. 
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Figure 4: The web graph optimized under discrete uniform transitions con- 
straints. In this case, the optimized graph has almost all internal links (links 
from a controlled page to another controlled page), so, for more readability, 
we display its adjacency matrix. The hyperlinks correspond to blue dots, 
obligatory links correspond to squares. The pages are ordered by decreas- 
ing average reward before teleportation (Section [s]). The optimal strategy 
consists in adding a lot of internal links excluding certain pages, as will be 
explained by the master Page theorem below (Theorem pi). 
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Proof. The set {v st ai^ + (1 — a)z G Vi} is a set of probability measures so 
it is clear that T is a-contr acting. Let w be its fixed point. For all i G [n], 



Wi = max ai'{ri .+w) + {1 — a)z-ri . = ma,xh'{ri .+w) — {l — a)zw 

V St aj/+{l-a)zePi ' ' v&^i 



We get equation (13) with constant (1 — ol)zw. □ 



Remark 7. T is the dynamic programming operator of a total reward dis- 
counted problem with discount rate a and rewards r- ■ = rij+^^-^ Sie[n] 
for transition from i to j (cf . Remark [s]) . 

Remark 8. The fixed point found is just the mean reward before telepor- 
tation at the optimum (see Definition |4j Section [s]) . 



We can then solve the dynamic programming equation (13) and so the 
PageRank Optimization Problem ([2]) or ([3]) with local constraints by value 
iteration. 

The algorithm starts with an initial potential function w, scans repeat- 
edly the pages and updates Wi when i is the current page according to 
Wi Ti{w) until convergence is reached. Then {w, (1 — a)zw) is solution 



of the ergodic dynamic programming equation (13) and the optimal linkage 
strategy is recovered by selecting the maximizing v at each page. 

Thanks to the damping factor a, the iteration can be seen to be a- 
contracting. Thus the algorithm converges in a number of steps independent 
of the dimension of the web graph. 

For the evaluation of the dynamic programming operator, one can use 
a linear program using to the description of the actions by facets. It is 
however usually possible to develop algorithms much faster than linear pro- 
gramming. We describe here a greedy algorithm for the discrete PageRank 
Optimization problem. The algorithm is straightforward if the set of oblig- 
atory links Oi is empty (Propositions [3] and |4]), so we only describe it in the 
other case. In Algorithm [T| J represents the set of facultative hyperlinks 
activated. We initialize it with the empty set and we augment it with the 
best hyperlink until it is not valuable any more to add a hyperlink. 

Proposition 8. When the constraints of the Discrete PageRank Optimiza- 
tion problem ^ are defined by obligatory, facultative and forbidden links, 
the greedy algorithm (Algorithm^ started at page i returns Ti{w) as defined 
in Proposition^ 

Proof. The local constraints are obviously respected by construction. At the 
end of the loop, we have the best choice of facultative outlinks from page i 
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Algorithm 1 Evaluation of the dynamic programming operator in the dis- 
crete problem 
1: Initialization: J and A; 1 

2: Sort (wi + ri^i)ifzjr. in decreasing order and let ^ : {1, . . . , \J^i\} — )• J^i be 

the sort function so that + rj^^(i) > • • • > u'^dj-.i) + ri^-,p{\T,\)- 

3: while jjiq^ E/eJuo,(^' + ^^,0 < ^i^ik) + ^iMk) ^nd k < do 
4: J ^ JU {V'(A;)} and A; ^ A; + 1 
5: end while 

6: Ti{w) = ajjj^^ EieJuoS^i + + (1 - a) EieH 



with exactly | J| outlinks. But as | YlieJuoS^'' ~'~ — ~'~ 

jj\^\EieJuoS'^i + ^ \J\+\k\+i ^leJuoMJ}^'^^ +'^^'')' 

implies that we have the best choice of outlinks. □ 

Remark 9. A straightforward modification of the greedy algorithm can 
handle a upper or a lower limit on the number of links on a given page. 

Proposition 9. An e approximation of the Discrete PageRank Optimization 
Problem ([3| with only local constraints can be done in time 




Proof. The value of the PageRank optimization problem is (1 — a)zw where 
w = T{w). Thus it is bounded by (1 tt) II II 1 II — (1 o) Illy II oo • The 
greedy algorithm described in the preceding paragraph evaluates the ith 
coordinate of the dynamic programming operator T in a time bounded by 
OdOjl + |J^j| log(|J^j|)) (by performing a matrix-vector product and a sort). 
Thus it evaluates the dynamic programming operator in a time bounded by 
o(E^eH|0«l + l-^dlog(|-F,|)). 

Now, if we normalize the rewards and if we begin the value iteration 
with = 0, the initial error is less than 1 in sup-norm. The fixed point 
iteration reduces this error by at least a, so we have to find k gN such that 
a'' <e. With k > the result holds. □ 

— — log(a) ' 

This result should be compared to PageRank computation's complexity 
by the power method [5], which is O I]ie[n]|Cj| + |-^i|) • 



25 



5 General shape of an optimized web site 



We now use the previous model to identify the features of optimal link 
strategies. In particular, we shall identify circumstances under which there 
is always one "master" page, to which all other pages should link. 

As in the work of De Kerchove, Ninove and Van Dooren [H], we shall use 
the mean reward before teleportation to study the optimal outlink strategies. 

Definition 4. Given a stochastic matrix P, the mean reward before tele- 
portation is given by v{P) := {In — aS)~^f, where fj = Ptjrij. 

Recall that 5 is the original matrix (without damping factor). 

Proposition 10. Suppose the instantaneous reward rij only depends on the 
current page i (rij = r[). Denote v{P) be the mean reward before telepor- 
tation (Definition^. Then P is an optimal link strategy of the continuous 
PageRank Optimization problem ^ if and only if 

yi G \n\, Pi . S argmaxi/f(P) 

Proof. We have Pv{P) = v{P) — r' + Tr{P)r' . Thus, using ve = 1, the 
condition of the proposition is equivalent to \/i G [n],Vi{P) + Ti{P)r' = 
maxi/g-p. u{v{P) + r'-e). By Proposition |6| this means that v{P) is the bias 
of Equation (|13|) and that P is an optimal outlink strategy. □ 



Remark 10. Proposition 10 shows that if P is any optimal outlink strategy, 
at every page i, the transition probability Pj^. must maximize the same linear 
function. 

Remark 11. If two pages have the same constraint sets, then they have the 
same optimal outlinks, independently of their PageRank. This is no more 
the case with coupling constraints. 

For the discrete PageRank Optimization problem, we have a more precise 
result: 

Theorem 5 (Master Page). Consider the Discrete PageRank Optimization 
problem ^ with constraints defined by given sets of obligatory, facultative 
and forbidden links. Suppose the instantaneous reward rij only depends on 
the current page i (rij = r[). Let v be the mean reward before teleportation 
(Definition^ at the optimum. Then any optimal link strategy must choose 
for every controlled page i all the facultative links {i,j) such that vj > 
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Figure 5: Maximization of the sum of the PageRank values of the colored 
pages. Top: obligatory links; self links are forbidden; all other links are 
facultative. Bottom: bold arcs represent an optimal linking strategy. Page 
4 points to all other controlled pages and Page 1, the master page, is pointed 
to by all other controlled pages. No facultative link towards an external page 
is selected. 



and any combination of facultative links such that vj = . Moreover, all 
optimal link strategies are obtained in this way. 

In particular, every controlled page should point to the page with the 
highest mean reward before teleportation, as soon as it is allowed to. We 
call it the "master page". 

Proof. By Remark |8| we know that the mean reward before teleportation at 
the optimum is a fixed point of the dynamic programming operator. In par- 
ticular, it is invariant by the application of the greedy algorithm (Algo- 
rithm[T]). Moreover, by Proposition [7| the mean reward before teleportation 
at the optimum is unique. 

Thus, any optimal strategy must let the mean reward before teleporta- 
tion invariant by the greedy algorithm. When there is no obligatory link 
from page i, either a link is selected and Vi = avj + or no link is 
selected and Vi = ZkVk + i^i > otVj + r'- for all facultative link {i,j). 

When there is at least one obligatory link, from Line 3 of the greedy algo- 
rithm, we know that, denoting J the set of activated links, all the links {i,j) 
verifying | YlieJuOi + < + rnust be activated. This can be 

rewritten as Vj > because Vi = a | Z^/eJuOi ~^ '^i- 

Finally, activating any combination of the facultative links such that 
Vj = gives the same mean reward before teleportation. □ 

The theorem is illustrated in Example 2 (Section |4]) and Figure [5] 

Example 3. The following simple counter examples show respectively that 
the conditions that instantaneous rewards only depend on the current page 
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and that there are only local constraints are useful in the preceding theorem. 
Take a two pages web graph without any design constraint. Set a = 0.85, 

z = (0.5, 0.5) and the reward per click r 1 10 



2 2 



Then v = (39.7, 35.^ 



Page 2 should link to Page 1 but Page 1 should link to Page 2 because 
39.7 + 1 < 35.8 + 10. 

Take the same graph as in preceding example. Set r' = (0, 1) and the 
coupling constraint that vri > tt2. Then every optimal strategy leads to 
TTi = TT2 = 0.5. This means that there is no "master" page because both 
pages must be linked to in order to reach vTj = 0.5. 

Remark 12. If every controlled page is allowed to point to every page, 
as in Figures [3] and [4j there is a master page to which every page should 
point. Actually, knowing that the optimal solutions are degenerate might 
be of interest to detect link spamming (or avoid being classified as a link 



spammer). The result of Proposition 10 and Theorem pi can be related to [3], 



where the authors show various optimal strategies for link farms: patterns 
with every page linking to one single master page also appear in their study. 
We also remark that in [4j , the authors show that making collusions is a good 
way to improve PageRank. We give here the page with which one should 
make a collusion. 

Remark 13. If there exists a page with maximal reward in which all the 
hyperlinks can be changed, then this page is the master page. It will have 
a single hyperlink, pointing to the second highest page in terms of mean 
reward before teleportation. 

Remark 14. Major search engines have spent lots of efforts on crawling 
the web to discover web pages and the hyperlinks between them. They can 
thus compute accurately the PageRank. A search engine optimization team 
may not have such a database available. If one can program a crawler to get 
a portion of the web graph or download some datasets of reasonable size for 
free ([29J for instance), these are still incomplete crawlings when compared 
to the search engine's. 

We denote by v and v the mean reward before teleportation of respec- 
tively the search engine's web graph and the trucated web graph. Let I be 
the set of pages of interest, that is the pages containing or being pointed to 
by a facultative link. We denote by R the length of a shortest path from 
a page in I to an uncrawled page. We can easily show that if there are no 
page without outlink, then for all i in /, |fi — < a^"*"^ ]T^7^||^||oo• 
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When there are pages without outhnk, the problem is more techni- 
cal. A possible approach to deal with it is to use the non-compensated 
PageRank [30|. 



6 PageRank Optimization with coupling constraints 

6.1 Reduction of the problem with coupHng constraints to 
constrained Markov decision processes 

From now on, we have studied discrete or continuous PageRank Optimiza- 
tion problems but only with local constraints. We consider in this section 
the following PageRank Optimization problem ^ with ergodic (linear in 
the occupation measure) coupling constraints: 

i-PiJ^iJ St. 



ttP = 7r , TT e^n , Pi,- eVi,yi e[n] (16) 



E 



Examples of ergodic coupling constraints are given in Section 2.3 
When coupling constraints are present, the previous standard ergodic 
control model is no longer valid, but we can use instead the theory of con- 
strained Markov decision processes. We refer the reader to [3l] for more 
background. In addition to the instantaneous reward r, which is used to 
define the ergodic functional which is maximized, we now consider a fi- 
nite family of cost functions {d^)k<^K, together with real constants {V^)k^K, 
which will be used to define the ergodic constraints. The ergodic constrained 
Markov decision problem consists in finding an admissible control strategy 
{vt)t>o, vt G Axt.^t G> 0, maximizing: 

1 ^"^ 

liminf-E(J]r(Xi,i.i)) (17) 

t=0 

under the \K\ ergodic constraints 

T-1 

F. 

T->+oo T 



1 ^"^ 

limsup-E(^d'=(Xj,z^t)) < F^ \/k£K 



t=o 
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where the controlled process {Xt)t>Q is such that 



F{Xt+i=j\Xt,ut)=pij\Xui^t) 



Theorem 4.1 in [3T] shows that one can restrict to stationary Markovian 
strategies and Theorem 4.3 in the same book gives an equivalent formulation 



of the ergodic constrained Markov decision problem (17) as a linear program. 
When Ai = extr{Vi), r{i,a) = T^jeH^idaj, d''{i,a) = T.jeln]dljaj and 
p{j\i,a) = Qj (see Proposition [5]) , it is easy to see that this linear program 
is equivalent to: 

max { ^ Pijrij st: p GTZ and ^ Pijd^j <V'',yk£K] (18) 

i,j&[n] i,je[n] 

where TZ is the image of HieM ^* correspondence of Proposition jlj 

The set 7^ is a polyhedron, as soon as every Vi is a polyhedron (Proposi- 
tion [2j). 

Following the correspondence discussed in Proposition [T| we can see that 



the linear Problem ( 18 ) is just the reformulation of Problem ( |16[ ) in terms 
of occupation measures when we consider total income utility Q^. 

The last result of this section gives a generalization to nonlinear utility 
functions: 

Proposition 11. Assume that the utility function U can be written as 
U{P) = W{p) where W is concave, that the local constraints are convex 
in P and that the coupling constraints are ergodic. Then, the PageRank Op- 



timization problem (16) is equivalent to a concave programming problem in 
the occupation measure p, from which e-solutions can be found in polynomial 
time. 

Proof. From Proposition [2] we know that the set of locally admissible occu- 
pation measures is convex. Adding ergodic (linear in the occupation mea- 
sure) constraints preserves this convexity property. So the whole optimiza- 
tion problem is concave. Finally, Theorem 5.3.1 in |32j states that e-solutions 
can be found in polynomial time. □ 

In particular, the (global) optimality of a given occupation measure can 
be checked by the first order optimality conditions which are standard in 
convex analysis. 

Remark 15. Proposition [Tl] applies in particular if is a relative en- 
tropy utility function, ie W{p) = -^ij^in]Pij^og{pij/pij), where parame- 
ters pij > (the reference measure) are given. 
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If we choose to minimize the entropy function on the whole web graph, 
we recover the TrafficRank algorithm [33]. When we control only some of 
the hyperlinks whereas the weights of the others are fixed, the solution of 
the optimization problem gives the webmaster the weights that she should 
set to her hyperlinks in order to have an entropic distribution of websurfers 
on her website, interpreted as a fair distribution of websurfers. 

In the next section, we extend the first order optimality conditions to 
the formulation in probability transitions, in order to get a characterization 
of the optimal linking strategies in the constrained PageRank Optimization 
problem. 



6.2 Optimality condition 

The following shows that the mean reward before teleportation (Definition|4]) 
determines the derivative of the utility function. Recall that the tangent cone 
Txix) of the set X at point x is the closure of the set of vectors q such that 
X + tq £ X for t small enough. 

Proposition 12. The derivative of total utility function Q is such that for 
all Q G Tv{P), 

(Df/(P),g) = Yl(^j{P)+r,,,)7T,{P)Q,j 

where v{P) is the mean reward before teleportation, tt{P) is the invariant 
measure of P and {■, ■) is the standard (Frohenius) scalar product on n x n 
matrices. 

Proof We have U{P) = T.i^jT^i{P)Pijrij = irr and vr = vrP = Tr{aS + 
(1 — a)ez). As vre = 1, we have an explicit expression for vr as function 
of P: it{P) = (1 — a)z{In — P + {1 — a)ez)~^. The result follows from 
derivation of ■K{P)f. We need to derive a product, to derive an inverse 
{{T>{A I— )• = —A~^HA~^) and the expression of the mean reward 

before teleportation v{P) = {In — P + (1 — a)ez)~^f. □ 

The next theorem, which involves the mean reward before teleporta- 
tion, shows that although the continuous constrained pagerank optimization 
problem is non-convex, the first-order necessary optimality condition is also 
sufficient. 

Theorem 6 (Optimality Condition). Suppose that the sets Vi defining local 
constraints are all closed convex sets, that the coupling constraints are given 
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by the ergodic costs functions df' , k ^ K and that the utility function is total 
income utility. Denote he the admissible set and v{P) the mean reward 
before teleportation (Definition^. We introduce the set of saturated con- 
straints Ksat = {k ^ K\ Ylij dij'n'iPij = V'^} and we introduce the numbers 
D'^j = TTid^j + 7rd''{I - aS)~^eiPij. Then the tangent cone of at P is 

Tp,(P) = {q G riiGH Tp.(^v) I VA; G Ksat , {D\Q) < o} and P* G 
is the optimum of the continuous PageRank Optimization problem Q with 
ergodic coupling constraints if and only if: 

VQ G Tp4P*) , Yl + n,j)Qi,j < 

i,je[n] 

Proof. Let us consider the birational change of variables of Proposition [T] 
As aU the occupation measures considered are irreducible, its Jacobian is in- 
vertible at any admissible point. Thus, we can use the results of Section 6.C 
in [33 • Denote V = Uieln] ^i. ^it^ tangent cone Tp(P) = l\ie[n] Tp^l^'i,), 
and7^ = f-HV). WehaveT7^d(p) = |cj G Tn{p) \ VA; G Ksat , {d^a) < o} 

and Tpd(P) = {q G M"^" | Vf-^Q G T7^d(/-i(i^))}. 

G TT^4f-^iP)) first means that Vf-^Q G TTi{f~\P)) which 
can also be written as V fV f~^Q = Q G T-p(P). The second condition 
is V/c G Ksat,{d^^^f'^Q) < 0. As {f-^{P))i,j = Pij = T^iPij, we have 
{Vf^^Q)ij = Ylk,lQk,l{Pk,lTp: + T^kSik^ji)- Thanks to the expression the 

derivative of the utility function and of = TTiej{I — aS)~^ek both given 
in Proposition |12[ we get the expression stated in the theorem. 



By Proposition 11 , the PageRank optimization problem is a concave pro- 
gramming problem in p and so, the first order (Euler) optimality condition 
guarantees the global optimality of a given measure. Thus, every station- 
ary point for the continuous PageRank Optimization problem is a global 
maximum when written in transition probabilities also. □ 

6.3 A Lagrangian relaxation scheme to handle coupling con- 
straints between pages 



The PageRank Optimization Problem with "ergodic" coupling constraints ( 16 ) 
may be solved by off the shelve simplex or interior points solvers. However, 
such general purpose solvers may be too slow, or too memory consuming, 
to solve the largest web instances. 

The following proposition yields an algorithm that decouples the compu- 
tation effort due to complexity of the graph and due to coupling constraints. 
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Proposition 13. The PageRank Optimization problem with K "ergodic" 



coupling constraints (16) can he solved by a Lagrangian relaxation scheme, 



in which the dual function and one of its subgradient 

9iX) = max{r,p) - ^ Xki{d\ p) - V'^) 



pen 
89 



k&K 

rk 



{X) = (d\p*{X))-V' 



dXk 

are evaluated by dynamic programming and p*{X) is a maximizer of the 
expression defining 0{X). 

Proof. This is a simple application of Lagrange multipliers theory, see [35] 
Theorem 21 and Remark 33 for instance. Here we relax the coupling con- 



straints in the problem written with occupation measures (18). We solve 



the dual problem, namely we minimize the dual function 9 on M;^'. The 
value of this dual problem is the same as the value of the constrained primal 
problem and we can get a solution of the primal problem since there is no 
duality gap. □ 

We have implemented a bundle high level algorithm, in which the dual 
function is evaluated at each step by running a value iteration algorithm, 
for a problem with modified reward. By comparison with the unconstrained 
case, the execution time is essentially multiplied by the number of iterations 
of the bundle algorithm. 



7 Experimental results 

7.1 Continuous problem with local constraints only 

We have tried our algorithms on a crawl on eight New Zealand Universities 
available at [36]. There are 413,639 nodes and 2,668,244 links in the graph. 
The controlled set we have chosen is the set of pages containing "maori" in 
their url. There are 1292 of them. We launched the experiments in a se- 
quential manner on a personal computer with Intel Xeon CPU at 2.98 Ghz 
and wrote the code in Scilab language. 

Assume that the webmasters controlling these pages cooperate and agree 
to change at most 20% of the links' weight to improve the PageRank, be- 
ing understood that self- links are forbidden (skeleton constraint, see Sec- 
tion 2.3). The algorithm launched on the optimization of the sum of the 



PageRanks of the controlled pages (calculated with respect to the crawled 
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graph only, not by the world wide graph considered by Google) ran 27 sec- 
onds. 

The optimal strategy returned is that every controlled page except it- 
self should link with 20% weight to maori-oteha.inassey.ac.nz/te_waka.htm. 
That page should link to maori-oteha.massey . ac .nz/tewaka/about .htm. The 
sum of PageRank values goes from 0.0057 to 0.0085. 

Hence, by uniting, this team of webmasters would improve the sum of 
their PageRank scores of 49%. Remark that all the pages point to the same 
page (except itself because self- links are forbidden). The two best pages 
to point to are in fact part of a "dead end" of the web graph containing 
only pages with maximal reward. A random surfer can only escape from 
this area of the graph by teleporting, which makes the mean reward before 
teleportation maximal. 



7.2 Discrete problem 

On the same data set, we have considered the discrete optimization problem. 
The set of obligatory links is the initial set of links. We have then selected 
2,319,174 facultative links on the set of controlled pages of preceding section. 

Execution time took 81 seconds with the polyhedral approach of Sec- 
tion |4.3| (60 iterations). We compared our algorithm with an adaptation 
of the graph augmentation approach of [T7] to total utility: this algorithm 
took 460 seconds (350 iterations) for the same precision. The optimal strat- 
egy is to add no link that goes out of the website but get the internal link 
structure a lot denser. From 12,288 internal links, the optimal strategy is 
to add 962873 internal links. Finally, 98.2% of the links are internal links 
and there is a mean number of links per page of 770. The sum of PageRank 
values jumps from 0.0057 to 0.0148. 

Here, as the weights of the links cannot be changed, the webmaster can 
hardly force websurfers to go to dead ends. But she can add so many links 
that websurfers get lost in the labyrinth of her site and do not find the 
outlinks, even if they were obligatory. 



7.3 Coupling linear constraints 

We would like to solve the discrete optimization problem of the preceding 
section with two additional coupling constraints. We require that each visi- 
tor coming on one of the pages of the team has a probability to leave the set 
of pages of the team on next step of 40% (coupling conditional probability 
constraint, see Section 2.3). We also require that the sum of PageRank val- 
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ues of the home pages of the 10 universities considered remains at least equal 
to their initial value after the optimization (effective frequency constraint). 

In the case of constrained Markov decision processes, optimal strategies 
are usually randomized strategies. This means that the theory cannot di- 
rectly deal with discrete action sets. Instead, we consider the continuous 
problem with the polytopes of uniform transition measures as local admissi- 
ble sets, i.e. we relax the discrete pattern. Thus by the Lagrangian scheme 
of Proposition 13, we get an upper bound on the optimal objective and we 
have a lower bound for any admissible discrete transition matrix. 

The initial value is 0.0057 and the Lagrangian relaxation scheme gives 
an upper bound of 0.00769. Computation took 675 s (11 high level it- 
erations). During the course of the Lagrangian relaxation scheme, all in- 
termediate solutions are discrete and three of them satisfied the coupling 
constraints. The best of them corresponds to a sum of PageRank values of 
0.00756. Thus we have here a duality gap of at most 1.7%. In general, the 
intermediate discrete solutions need not satisfy the coupling constraints and 
getting an admissible discrete solution may be difficult. 

The discrete transition matrix found suggests to add 124,328 internal 



links but also 11,235 external links. As in Section 7.2, lots of links are 
added, but here there are also external links. 

The bounding technique proposed here can also be adapted to PageRank 
optimization problem with mutual exclusion constraints. It may also be 
possible to use it to design a branch and bound algorithm to solve the 
problem exactly thanks to the bounds found. 



Conclusion 

We have presented in this paper a general framework to study the opti- 
mization of PageRank. Our results apply to a continuous problem where 
the webmaster can choose the weights of the hyperlinks on her pages and 
to the discrete problem in which a binary decision must be taken to decide 
whether a link is present. We have shown that the Discrete PageRank Op- 
timization problem without coupling constraints can be solved by reduction 
to a concisely described relaxed continuous problem. We also showed that 
the continuous Pagerank optimization problem is polynomial time solvable, 
even with coupling constraints. 

We gave scalable algorithms which rely on an ergodic control model and 
on dynamic programming techniques. The first one, which applies to prob- 
lems with local design constraints, is a fixed point scheme whose convergence 
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rate shows that optimizing PageRank is not much more compHcated than 
computing it. The second algorithm, which handles coupling constraints, is 
still efficient when the number of coupling constraints remains small. 

We have seen that the mean reward before teleportation gives a total 
order of preference in pointing to a page or an other. This means that pages 
high in this order concentrate many inlinks from controlled pages. This is 
a rather degenerate strategy when we keep in mind that a web site should 
convey information. Nevertheless, the model allows to address more complex 
problems, for instance with coupling constraints, in order to get less trivial 
solutions. 

This work may be useful to understand link spamming, to price internet 
advertisements or, by changing the objective function, to design web sites 
with other goals like fairness or usefulness. The latter is the object of further 
research. 
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